Interpretation of Natural-language Robot Instructions: Probabilistic Knowledge Representation, Learning, and Reasoning

A robot that can be simply told in natural language what to do -- this has been one of the ultimate long-standing goals in both Artificial Intelligence and Robotics research. In near-future applications, robotic assistants and companions will have to understand and perform commands such as set the table for dinner'', make pancakes for breakfast'', or cut the pizza into 8 pieces.'' Although such instructions are only vaguely formulated, complex sequences of sophisticated and accurate manipulation activities need to be carried out in order to accomplish the respective tasks. The acquisition of knowledge about how to perform these activities from huge collections of natural-language instructions from the Internet has garnered a lot of attention within the last decade. However, natural language is typically massively unspecific, incomplete, ambiguous and vague and thus requires powerful means for interpretation. This work presents PRAC -- Probabilistic Action Cores -- an interpreter for natural-language instructions which is able to resolve vagueness and ambiguity in natural language and infer missing information pieces that are required to render an instruction executable by a robot. To this end, PRAC formulates the problem of instruction interpretation as a reasoning problem in first-order probabilistic knowledge bases. In particular, the system uses Markov logic networks as a carrier formalism for encoding uncertain knowledge. A novel framework for reasoning about unmodeled symbolic concepts is introduced, which incorporates ontological knowledge from taxonomies and exploits semantically similar relational structures in a domain of discourse. The resulting reasoning framework thus enables more compact representations of knowledge and exhibits strong generalization performance when being learnt from very sparse data. Furthermore, a novel approach for completing directives is presented, which applies semantic analogical reasoning to transfer knowledge collected from thousands of natural-language instruction sheets to new situations. In addition, a cohesive processing pipeline is described that transforms vague and incomplete task formulations into sequences of formally specified robot plans. The system is connected to a plan executive that is able to execute the computed plans in a simulator. Experiments conducted in a publicly accessible, browser-based web interface showcase that PRAC is capable of closing the loop from natural-language instructions to their execution by a robot

Joint Probability Trees

We introduce Joint Probability Trees (JPT), a novel approach that makes learning of and reasoning about joint probability distributions tractable for practical applications. JPTs support both symbolic and subsymbolic variables in a single hybrid model, and they do not rely on prior knowledge about variable dependencies or families of distributions. JPT representations build on tree structures that partition the problem space into relevant subregions that are elicited from the training data instead of postulating a rigid dependency model prior to learning. Learning and reasoning scale linearly in JPTs, and the tree structure allows white-box reasoning about any posterior probability $P(Q|E)$, such that interpretable explanations can be provided for any inference result. Our experiments showcase the practical applicability of JPTs in high-dimensional heterogeneous probability spaces with millions of training samples, making it a promising alternative to classic probabilistic graphical models

Recapitulation of tumor heterogeneity and molecular signatures in a 3D brain cancer model with decreased sensitivity to histone deacetylase inhibition

INTRODUCTION Physiologically relevant pre-clinical ex vivo models recapitulating CNS tumor micro-environmental complexity will aid development of biologically-targeted agents. We present comprehensive characterization of tumor aggregates generated using the 3D Rotary Cell Culture System (RCCS). METHODS CNS cancer cell lines were grown in conventional 2D cultures and the RCCS and comparison with a cohort of 53 pediatric high grade gliomas conducted by genome wide gene expression and microRNA arrays, coupled with immunohistochemistry, ex vivo magnetic resonance spectroscopy and drug sensitivity evaluation using the histone deacetylase inhibitor, Vorinostat. RESULTS Macroscopic RCCS aggregates recapitulated the heterogeneous morphology of brain tumors with a distinct proliferating rim, necrotic core and oxygen tension gradient. Gene expression and microRNA analyses revealed significant differences with 3D expression intermediate to 2D cultures and primary brain tumors. Metabolic profiling revealed differential profiles, with an increase in tumor specific metabolites in 3D. To evaluate the potential of the RCCS as a drug testing tool, we determined the efficacy of Vorinostat against aggregates of U87 and KNS42 glioblastoma cells. Both lines demonstrated markedly reduced sensitivity when assaying in 3D culture conditions compared to classical 2D drug screen approaches. CONCLUSIONS Our comprehensive characterization demonstrates that 3D RCCS culture of high grade brain tumor cells has profound effects on the genetic, epigenetic and metabolic profiles of cultured cells, with these cells residing as an intermediate phenotype between that of 2D cultures and primary tumors. There is a discrepancy between 2D culture and tumor molecular profiles, and RCCS partially re-capitulates tissue specific features, allowing drug testing in a more relevant ex vivo system

Interpretation Natürlichsprachlicher Roboterinstruktionen : Repräsentieren, Lernen und Schlussfolgern von Probabilistischem Wissen

A robot that can be simply told in natural language what to do -- this has been one of the ultimate long-standing goals in both Artificial Intelligence and Robotics research. In near-future applications, robotic assistants and companions will have to understand and perform commands such as set the table for dinner'', make pancakes for breakfast'', or cut the pizza into 8 pieces.'' Although such instructions are only vaguely formulated, complex sequences of sophisticated and accurate manipulation activities need to be carried out in order to accomplish the respective tasks. The acquisition of knowledge about how to perform these activities from huge collections of natural-language instructions from the Internet has garnered a lot of attention within the last decade. However, natural language is typically massively unspecific, incomplete, ambiguous and vague and thus requires powerful means for interpretation. This work presents PRAC -- Probabilistic Action Cores -- an interpreter for natural-language instructions which is able to resolve vagueness and ambiguity in natural language and infer missing information pieces that are required to render an instruction executable by a robot. To this end, PRAC formulates the problem of instruction interpretation as a reasoning problem in first-order probabilistic knowledge bases. In particular, the system uses Markov logic networks as a carrier formalism for encoding uncertain knowledge. A novel framework for reasoning about unmodeled symbolic concepts is introduced, which incorporates ontological knowledge from taxonomies and exploits semantically similar relational structures in a domain of discourse. The resulting reasoning framework thus enables more compact representations of knowledge and exhibits strong generalization performance when being learnt from very sparse data. Furthermore, a novel approach for completing directives is presented, which applies semantic analogical reasoning to transfer knowledge collected from thousands of natural-language instruction sheets to new situations. In addition, a cohesive processing pipeline is described that transforms vague and incomplete task formulations into sequences of formally specified robot plans. The system is connected to a plan executive that is able to execute the computed plans in a simulator. Experiments conducted in a publicly accessible, browser-based web interface showcase that PRAC is capable of closing the loop from natural-language instructions to their execution by a robot

Everything Robots Always Wanted to Know about Housework (But were afraid to ask)

Abstract — In this paper we discuss the problem of actionspecific knowledge processing, representation and acquisition by autonomous robots performing everyday activities. We report on a thorough analysis of the household domain which has been performed on a large corpus of natural-language instructions from the Web, which underlines the supreme need of action-specific knowledge for robots acting in those environments. We introduce the concept of Probabilistic Robot Action Cores (PRAC) that are well-suited for encoding such knowledge in a probabilistic first-order knowledge base. We additionally show how such a knowledge base can be acquired by natural language and we address the problems of incompleteness, underspecification and ambiguity of naturalistic action specifications and point out how PRAC models can tackle those. I

RoboSherlock: Unstructured information processing for robot perception

We present ROBOSHERLOCK, an open source software framework for implementing perception systems for robots performing human-scale everyday manipulation tasks. In ROBOSHERLOCK, perception and interpretation of realistic scenes is formulated as an unstructured information management (UIM) problem. The application of the UIM principle supports the implementation of perception systems that can answer task-relevant queries about objects in a scene, boost object recognition performance by combining the strengths of multiple perception algorithms, support knowledge-enabled reasoning about objects and enable automatic and knowledge-driven generation of processing pipelines. We demonstrate the potential of the proposed framework by three feasibility studies of systems for real-world scene perception that have been built on top of ROBOSHERLOCK